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Abstract 

Today,  rapid  product  innovation  is  essential  to  remain  competitive.  To  help  spur  innovation 
in  the  acquisition  of  major  defense  systems  and  reduce  their  cost,  time  and  risks,  the  Department 
of  Defense  launched  the  Computational  Research  and  Engineering  Acquisition  Tools  and 
Environments  (CREATE)  Program  in  2006.  The  CREATE  goal  is  to  develop  and  deploy 
physics-based  high  performance  computing  software  applications  for  the  design  and  analysis  of 
military  air  craft,  ships,  and  radio  frequency  antenna  systems  (and  more  recently  ground 
vehicles)  through  the  construction  and  analysis  of  virtual  prototypes  for  those  systems.  Code 
development  began  in  2008,  and  eight  years  later,  CREATE  is  already  beginning  to  accomplish 
these  goals. 

Introduction 

Since  the  end  of  World  War  II,  the  calculating  power  of  computers  has  grown  exponentially 
from  ~1  Floating  Point  Operation  /  second  (FLOP/s)  to  over  10 16  FLOP/s 
(http://www.top500.org/).  This  means  that— for  the  first  time  in  history— we  have  the  potential  to 
make  accurate  predictions  of  the  behavior  of  complex  physical  systems  (e.g.  the  weather,  the 
behavior  of  chemical  systems,  airplanes,  ships,  automobiles,  etc.).  We  can  now  (2015)  develop 
and  deploy  science-based  software  applications  for  high  performance  computers  that: 

1 .  Include  the  major  physical  effects  that  detennine  the  performance  of  the  system, 

2.  Utilize  highly  accurate  mathematical  and  numerical  solution  algorithms, 

3.  Are  experimentally  validated, 

4.  Can  predict  the  performance  of  a  full-scale  system  (e.g.  an  entire  ship  or  air  plane), 

5.  Enable  multidimensional  design  of  experiments  to  generate  large  trade-spaces  for  a  full 

scale  system,  and 

6.  Can  complete  a  high-fidelity,  time-dependent,  three-dimensional  multi-physics  calculation 

for  a  maneuvering  system  in  a  few  days  in  2015  that  took  weeks  in  2005,  and  months 
(if  even  possible  at  the  same  level  of  fidelity)  in  1995. 


1  The  main  authors  and  S.  Morton,  R.  Strawn,  J.  Livingston,  A.  Mackenna,  J.  Gorski,  E.  Moyer, 
L.  G.  Votta,  P.A.  Gibson,  D.  Borovitcky,  L.K.  Miller,  S.B.  Allwerdt,  and  the  approximately  120 

other  CREATE  staff. 
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This  capability  enables  design  engineers  to  construct  realistic  virtual  prototypes  of  physical 
systems  (ships,  microprocessors,  earth  moving  equipment,  etc.)  and  make  accurate  predictions  of 
their  performance  by  solving  the  physics  equations  that  govern  their  behavior. 

In  the  past,  it  was  necessary  to  construct  real  prototypes  and  use  live  tests  to  assess  their 
perfonnance  and  find  the  design  flaws.  With  simple  systems,  and  incremental  changes,  there  was 
time  to  follow  the  traditional  product  development  paradigm  of  “design,  build,  test,  fail,  re¬ 
design,. . .”  iterated  cycles.  For  the  standard  system  engineering  product  development  process[l] 
(Figure  1),  the  design  of  new  products  is  based  on  “rule  of  thumb”  extrapolations  of  existing 
products.  Sub-system  physical  prototypes  are  tested  during  the  engineering  design  phase,  and  full 
system  physical  prototypes  are  tested  just  before  full  scale  production.  With  today’s  more 
complex  systems,  these  live  tests  occur  too  late  to  provide  timely  data  on  design  defects  and 
perfonnance  shortfalls.  Expensive  and  time-consuming  rework  is  required  to  fix  the  problems 
uncovered  by  live  testing. 

In  2006,  recognizing  the  need  to  help  spur  innovation  for  major  defense  systems,  the 
Department  of  Defense  launched  the  Computational  Research  and  Engineering  Acquisition 
Tools  and  Environments  (CREATE)  Program.  CREATE  was  chartered  to  reduce  the  cost,  time 
and  risks  of  DoD  acquisition  programs  by  developing  and  deploying  multiphysics-based  high 
perfonnance  computing  software  applications  for  the  design  and  analysis  of  military  air  craft, 
ships,  and  radio  frequency  antenna  systems. 

Using  the  CREATE  tools,  engineers  can  construct  virtual  prototypes  and  analyze  product 
perfonnance  at  any  stage  of  the  development  process,  supplementing  data  from  live  tests.  For 
conceptual  design  of  new  systems,  “rule-of-thumb”  extrapolations  of  existing  designs  can  be 
replaced  with  physics-based  generation  of  design  options  allowing  rapid  trade-space  exploration, 
and  assessment  of  the  feasibility  of  design  options  with  physics-based  analysis  tools.  They  can 
consider  thousands  of  design  options  instead  of  a  few.  For  detailed  design  development,  high- 
fidelity  analysis  of  virtual  prototypes  can  replace  “failure  data  from  live  tests”  with  “physics- 
based  accurate  predictions  of  virtual  prototype  perfonnance.”  This  provides  timely  decision  data 
that  enables  engineers  to  identify  design  flaws  and  perfonnance  shortfalls  early,  allowing 
problems  to  be  fixed  before  metal  is  cut,  minimizing  rework  that  cause  schedule  delays  and  cost 
growth.  The  tools  can  be  applied  at  all  stages  of  the  product  development  process,  from  early 
concept  design  through  operation,  support,  sustainment  and  modernization. 

Code  development  began  in  2008,  and  eight  years  later,  CREATE  is  beginning  to  accomplish 
those  goals.  The  CREATE  tools  are  being  used  by  more  than  70  DoD  organizations  for  the 
assessment  of  more  than  80  DoD  weapon  systems  and  platforms.  There  are  over  600  active 
CREATE  user  software  licenses. 

The  CREATE  tools  can  help  make  the  testing  process  more  productive,  more  effective,  and 
more  efficient.  The  tools  can  be  used  to  identify  the  most  sensitive  and  uncertain  operating 
conditions  so  that  testing  programs  can  concentrate  on  those  areas,  so  that  the  total  test  data 
requirements  can  be  reduced  a  factor  of  5  or  more  with  a  concomitant  reduction  in  the  required 
testing  time[2].  In  addition  it  frees  the  testing  community  to  address  the  basic  issues  that 
detennine  weapon  system  perfonnance. 
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Figure  1.  System  Engineering  Product  Development  Life  Cycle  Model  (adapted  from  [1]) 

Many  industries  and  Federal  Agencies  have  successfully  adopted  the  virtual  prototype 
paradigm[3-5].  Until  recently,  however,  there  has  been  little  public  information  on  the  value  of 
the  paradigm,  chiefly  because  it  gives  organizations  that  use  it  a  significant  competitive 
advantage,  and  their  use  of  the  paradigm  is  considered  a  trade  secret.  A  few  examples  are 
publically  available.  The  Partnership  for  Advanced  Computing  in  Europe  (PRACE)  established 
an  Automotive  Simulation  Center  (ASC)  in  Stuttgart,  with  projects  in  vehicle  drive,  vehicle 
structure,  vehicle  physics,  etc.  ASC  director,  Alexander  Walser,  noted  in  2014  that  “numerical 
simulation  made  its  way  in  the  design  phase  of  automotive  development  and  productions  (as)  a 
useful  tool  for  faster  problem  analysis  and  reduction  of  cost  and  product  design  time. [6]” 

Specific  benefits,  however,  were  not  detailed.  In  an  aerospace  industry  example,  however,  Doug 
Ball,  then  Boeing’s  chief  engineer  for  enabling  technology  and  research,  stated  “When  we  were 
designing  the  767  back  in  the  1980s,  we  built  and  tested  about  77  wings.  By  using 
supercomputers  to  simulate  the  properties  of  the  wings  on  recent  models  such  as  the  787  and  the 
747-8,  we  only  had  to  design  seven  wings,  a  tremendous  savings  in  time  and  cost. .  .”[7] 

Starting  in  1992,  Goodyear  developed  a  physics-based  design  tool  for  tires  that  became 
the  core  of  their  “innovation  engine. ’’[4]  Goodyear  CEO  Richard  Kramer  noted  in  the  Q4  2010 
Earnings  Call:  “Our  innovation  engine  again  delivered  in  2010.  The  percentage  of  new  products 
in  our  overall  lineup  is  the  highest  ever. . .  Our  innovative  new  products  continue  to  accumulate 
an  impressive  list  of  test  wins  and  third-party  endorsements.’’[8]  In  the  2009  Annual  Report,  then 
Goodyear  CEO  Robert  Keagan  stated  that:  “Our  new  product  engine  is  poised  to  take  advantage 
of  the  demand  for  high-value-added  tires  and  to  do  so  with  unmatched  speed  to  market.”[9]  With 
this  approach,  Goodyear  reduced  its  product  development  time  from  three  years  to  as  little  as 
eight  months  and  reduced  its  new  product  prototyping  and  testing  costs  from  40%  to  15%  of  the 
R&D  budget ,  an  annual  savings  of  $100  million.  [10] 

The  CREATE  Program 

CREATE  began  in  2006  as  a  set  of  four  projects  (Air  Vehicles,  Naval  Ships,  Radio 
Frequency  Antennas,  and  Meshing  and  Geometry)  with  nine  individual  software  products  (Table 
1).  The  choice  of  the  projects  was  dictated  by  consideration  of  the  relative  size  and  importance  of 
major  defense  acquisition  programs  and  the  potential  of  physics-based  high  performance 
computing  applications  to  predict  the  performance  of  those  systems. 
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Each  of  the  CREATE  projects  has  two  types  of  software  products:  1)  a  concept  development 
tool  to  generate  conceptual  designs  and  analyze  their  feasibility  and  performance  using  fast,  but 
lower-fidelity  tools  (DaVinci,  RSDE/IHDE,  and  a  module  of  SENTRi);  and  2)  high-fidelity  tools 
to  provide  accurate  predictions  of  the  system  performance  (Kestrel,  Helios,  NESM,  NavyFOAM, 
and  SENTRi).  A  fourth  project,  Meshing  and  Geometry,  provides  capability  for  concept  design 
tools  to  generate  numerical  representations  of  the  platform  and  then  to  generate  the  meshes 
needed  for  computational  analysis  (Capstone).  The  Meshing  and  Geometry  tool  can  also  produce 
meshes  from  geometry  representations  generated  by  the  most  commonly  used  commercial  CAD 
tools.  Overview  papers  describing  these  products  will  appear  in  this  and  future  issues  of  this 
journal. 


Table  1:  HPCMP  CREATE™  Projects  and  Products 

•  Air  Vehicles— CREATE  AV 

-  DaVinci  -  Rapid  conceptual  design 

-  Kestrel  -  High-fidelity,  full-vehicle,  multi-physics  analysis  tool  for  fixed-wing 
aircraft 

-  Helios  -  High-fidelity,  full-vehicle,  multi-physics  analysis  tool  for  rotary-wing 
aircraft 

•  Naval  Ships — CREATE  Ships 

-  Rapid  Ship  Design  Environment  (RSDE)  -  Rapid  Design  and  Synthesis  Capability 
—  Navy  Enhanced  Sierra  Mechanics  (NESM)  -  Ship  Shock  &  Shock  Damage 

Assessment 

—  NavyFOAM  -  Hydrodynamic-predict  hydrodynamic  performance 
—  Integrated  Hydro  Design  Environment  (IHDE)  -  Facilitates  hydrodynamic 
perfonnance  evaluations  in  early  stage  ship  design 

•  RF  Antenna — CREATE  RF 

—  SENTRi  -  Electromagnetics  antenna  design  integrated  with  platfonns 

•  Meshing  and  Geometry — CREATE  MG 

—  Capstone  -  Components  for  generating  geometries  and  meshes  needed  for  analysis 

The  CREATE  tools  have  been  developed,  deployed  and  supported  by  the  DoD  (DoD 
employees  and  contractors),  are  validated  with  DoD  experimental  data  for  DoD  use  cases,  and 
are  “owned”  by  the  DoD.  In  general  the  DoD  has  unlimited  distribution  rights  to  the  CREATE 
software.  The  CREATE  tools  give  DoD  acquisition  program  engineers  the  ability  to  make 
independent  assessments  of  proposed  designs  and  contractor  deliverables.  The  tools  are  directly 
helping  the  DoD  acquisition  engineering  community  grow  its  organic  engineering  capability,  a 
DoD  priority. 

Based  on  the  success  of  the  original  CREATE  projects,  in  2013  CREATE  began  initial 
development  of  a  fifth  project  for  the  design  and  analysis  of  Ground  Vehicles.  It  is  too  new  to 
have  yet  delivered  an  initial  release,  and  will  be  discussed  in  a  subsequent  paper. 

The  Nine  HPCMP  CREATE™  Products: 
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DaVinci — Concept  design  tool  for  air  vehicles: 

DaVinci  will  allow  engineers  to  populate  design  option  spaces  of  fixed  and  rotary  wing 
aircraft  and  provide  an  initial  assessment  of  the  performance  of  the  design.  Choosing  from 
previously  engineered  components,  the  tool  will  allow  engineers  to  select  and  modify  wings, 
fuselage,  and  propulsion  components.  Aircraft  designers  then  can  select,  adjust,  and  rearrange 
internal  components  in  aircraft  designs.  It  is  “Model  Centric”.  It  will  support  highly  efficient 
construction  and  maintenance  of  Air  Vehicle  models,  including  geometry  that  is  parametric  and 
includes  water-tight  external  geometry,  internal  structure,  subsystem  layout,  volumes,  and  mass 
properties.  It  will  provide  a  multi-disciplinary,  physics-based  analysis  capability  that  is  variable 
fidelity,  but  is  consistent  across  disciplines.  It  provides  the  ability  to  persist  design  data  and 
intent  throughout  acquisition  and  program  life.  The  virtual  model  can  be  tested  virtually  through 
its  flight  envelope  to  assess  basic  performance  characters,  including  mission  performance,  and 
decision  support  with  uncertainty  quantification  and  sensitivity  analysis. 

The  current  version  (3.1)  has  the  capability  to  build  parametric  designs  for  fighter, 
transport,  and  surveillance  aircraft  that  include  the  Outer  Mold  Line  (OML)  and  structural 
envelope.  It  supports  geometric  analysis  including  areas,  volumes,  centroids  and  moments  and 
high  fidelity  aerodynamic  analysis  using  the  high-fidelity  tool  Kestrel.  It  can  design  and  build  a 
geometric  model  of  a  platfonn,  and  build  a  mesh  that  captures  this  geometry  with  Capstone  for 
high  fidelity  analysis  with  Kestrel.  Surrogate  geometry  models  have  been  built  for  the  support  of 
the  KC-46,  the  new  Air  Force  Tanker  based  on  the  Boeing  767,  and  JSTARS,  the  new  Air  Force 
AWACS  airplane  based,  for  instance,  on  a  Gulfstream  650,  550  or  a  Boeing  737. 

Kestrel — High-Fidelity  Tool  for  Fixed  Wing  Air  Vehicle  Performance  Prediction 

Kestrel  has  the  capability  to  provide  accurate  predictions  of  the  perfonnance  of  DoD  air 
vehicles,  with  a  specific  focus  on  the  fixed-wing  community.  It  integrates  computational  fluid 
dynamics,  structural  dynamics,  propulsion,  and  control  for  sub-sonic  through  supersonic  aircraft 
operation.  In  detail,  the  capabilities  available  in  Kestrel  v5.0  include:  1)  Aerodynamics  (Navier- 
Stokes  solvers  and  a  full  suite  of  boundary  conditions  and  turbulence  models),  2)  Structural 
Dynamics  (Modal  models  or  Finite  Element  Analysis  for  aero-structure  interactions),  3)  Flight 
Control  Systems  (Control  surface  movement  -  defonning  geometry  or  overset),  and  4) 

Propulsion  (Engine  “cycle-decks”  for  propulsion  effects,  or  direct  engine  simulation  including 
inlet  and  rotating  machinery,  nozzle,  and  moving  walls).  This  set  of  capabilities  is  unique  in  the 
international  aerospace  community.  It  provides  the  capability  to  develop  major  innovations  in  the 
design  of  next  generation  aeronautical  weapon  systems.  With  Kestrel,  engineers  can  verify 
designs  prior  to  key  decision  points  (and  prior  to  fabrication  of  test  articles  or  full-scale 
prototypes),  plan  and  rehearse  wind-tunnel  and  full-scale  flight  tests,  evaluate  planned  (or 
potential)  operational  use  scenarios,  perform  flight  certifications  (e.g.,  airworthiness,  flight 
envelope  expansion,  mishap  investigation,  etc.),  and  generate  response  surfaces  usable  in 
DaVinci,  flight-simulators,  and  other  environments  that  require  real-time  access  to  perfonnance 
data.  Kestrel  has  been  applied  to  the  analysis  of  over  30  fixed-wing  DoD  Aviation  systems 
including  store  separation,  A-10,  F-18E,  F-15,  B-52,  E-2D,  P-3,  and  many  others. 

Helios — High-Fidelity  Tool  for  Rotary  Wing  Air  Vehicle  Performance  Prediction 

Helios  is  a  high-fidelity,  full-vehicle,  multi-physics  analysis  tool  for  rotary-wing  aircraft. 
Helios  v5.0  can  calculate  the  performance  of  a  full  sized  rotorcraft,  including  the  fuselage  and 
rotors.  It  can  handle  arbitrary  rotor  configurations  (e.g.,  conventional  main  rotor/tail-fan;  co-axial 
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main  rotor/pusher  propeller;  tandem  main  rotors;  tiltrotors;  quad-tiltrotors;  etc).  It  has  the 
capability  to  analyze  and  predict  prescribed  maneuvers  with  tight  coupling  of  rotor  aero- 
structural  dynamics.  A  highly  accurate  treatment  of  the  vortex  shedding  from  the  rotor  blade  tips 
using  adaptive  mesh  refinement  gives  Helios  unique  capability  to  assess  the  interaction  of  these 
vortices  with  the  fuselage  and  nearby  rotor  blades.  Helios  can  provide  all  the  benefits  for  rotary¬ 
winged  aircraft  that  Kestrel  can  for  fixed-wing  aircraft. 

There  have  already  been  important  examples  of  the  use  and  value  of  Helios.  The  Army 
Rotorcraft  Program  (AMRDEC/AED)  used  Helios  with  Boeing  to  generate  early  design  stage 
predictions  of  helicopter  performance  for  a  proposed  rotor  blade  upgrade  for  the  CH-47F 
helicopter  (Chinook)  to  achieve  up  to  an  estimated  2,000  pounds  improved  hover  thrust  for  400+ 
Chinooks  with  limited  degradation  of  forward  flight  performance.  The  Army  Joint-Multi-Role 
Technology  Demonstrator  (JMR-TD)  Program  used  Helios  to  provide  decision  data  on  the 
proposals  from  four  vendors  for  the  JMR-TD  program.  Helios  enabled  government  engineers  to 
provide  the  government  the  ability  to  conduct  an  independent  analysis  of  the  contractor 
proposals.  The  Army  Rotorcraft  Program  (AMRDEC/AED)  is  using  Helios  to  assess  the  H-60 
tail  rotor  effectiveness  for  providing  directional  control  of  aircraft  in  combination  with  increased 
engine  power  and  main  rotor  performance. 

Rapid  Ship  Design  Environment  (RSDE)  — Rapid  Concept  Design  for  Ships: 

RSDE  is  a  concept  design  tool  that  allows  engineers  and  naval  architects  to  assess  the 
trade-offs  inherent  in  designing  ships  to  meet  a  spectrum  of  competing  key  performance 
parameters.  Employing  the  concept  of  design  space  exploration,  engineers  and  naval  architects 
can  provide  data  for  decision  makers  on  the  impact  of  tradeoffs  in  range,  speed,  armament, 
aviation  support,  etc.  on  the  size  and,  in  large  measure,  the  cost  of  a  proposed  ship  concept. 
RSDE  can  generate  tens  of  thousands  of  candidate  ship  designs  with  varying  hullforms, 
subdivision,  and  machinery  arrangements.  An  initial  assessment  of  the  intact  and  damaged 
stability,  resistance,  and  an  initial  structural  design  and  analysis  is  done  for  each  candidate  ship 
design.  RSDE  has  been  used  to  enable  set-based  design[l  1]  on  Navy  acquisition  programs.  This 
design  method  allows  down-selection  of  a  ship  design  to  occur  later  in  the  process  when  the 
trade-offs  are  more  fully  understood.  It  has  been  applied  to  numerous  ship  design  studies 
including  the  Amphibious  Landing  Craft  LX(R)  Analysis  of  Alternatives,  and  the  Small  Surface 
Combatant  Trade  Study.  The  team  leader  for  the  RSDE  tool  was  awarded  the  2014  American 
Society  of  Naval  Engineers  (ASNE)  Gold  Medal  for  his  work  developing  and  applying  the 
RSDE  tool. 

NavyFOAM  -  High  fidelity  predictions  of  ship  hydrodynamic  performance 

NavyFOAM  is  based  on  the  OpenFOAM  (www.openfoam.orq)  libraries  and  code 
architecture.  To  that  base,  we  have  added  a  number  of  features  and  capabilities  that  enable 
simulation  of  the  air-sea  interface  (e.g.,  surface  waves)  and  other  effects  important  for  naval 
vessels.  NavyFOAM  is  a  fully  parallelized,  multi-physics  computational  fluid  dynamics  (CFD) 
framework  developed  using  modem  object-oriented  programming  (OOP).  The  code  enables 
high-fidelity  hydrodynamic  analysis  and  prediction  of  ship  performance  such  as  resistance, 
propulsion,  maneuvering,  seakeeping  and  seaway  loads.  It  has  demonstrated  accuracy  against 
experimental  data  for  a  number  of  target  applications  such  as  resistance,  propeller  characteristics, 
hull/propulsor  interaction,  and  6-degree-of-freedom  ship  motion  of  underwater  vehicles  and 
surface  ships.  Offering  a  suite  of  Navier-Stokes-based  flow  solvers  tailored  to  specific 
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applications  including  single-  and  multi-phase  solvers,  NavyFOAM  allows  assessment  of 
alternative  hull  and  propulsor  designs.  With  NavyFOAM  users  can  evaluate  a  ship’s 
performance  in  a  wide  array  of  operating  conditions  including  both  subsea  and  surface 
operations.  Its  modularity  expedites  coupling  with  third-party  software  and  collaborative  multi¬ 
disciplinary  software  development  (e.g.  fluid- structure  interaction,  hydroacoustics).  It  has  been 
applied  to  many  naval  systems  including  assessment  of  the  safe  operating  envelope  of  the  DDG- 
1000,  propeller  designs,  the  USMC  Amphibious  Combat  Vehicle,  the  Ohio  Replacement 
Program  (a  $100B  procurement  of  the  next  generation  ballistic  missile  launch  submarine)  and 
many  other  systems  of  interest  to  the  US  Navy. 

Navy  Enhanced  Sierra  Mechanics  (NESM)-  Ship  Shock  &  Shock  Damage  Assessment 

NESM  builds  on  the  Department  of  Energy’s  shock  analysis  tool  Sierra  Mechanics  to 
provide  a  means  to  assess  ship  and  component  response  to  external  shock  and  blast  using 
accurate  high  performance  computational  tools.  NESM  can  reduce  the  time  and  expense  required 
for  physical  shock  testing  of  ship  classes  and  also  improves  the  initial  ship  design  process  by 
assessing  planned  component  installations  for  shock  performance  prior  to  final  arrangement  and 
installations  decisions.  The  tightly  coupled  multi-physics  capabilities  include:  1)  Structural 
Dynamics  (Implicit  linear-elastic  solver:  static,  modal,  transient,  acoustics,  and  more),  2)  Solid 
Mechanics  (Explicit  plasticity  solver:  failure,  high-strain,  multi-grid,  and  more),  3)  Fluid 
Dynamics  (Euler  solver:  shock  propagation,  load  environments,  and  threat  modeling),  and  4) 
Fluid-Structure  Interaction.  The  solution  algorithms  in  NESM  can  exploit  massively  parallel 
computers,  and  can  scale  to  thousands  of  cores,  enabling  efficient  computer  use  and  the  ability  to 
address  full-sized  naval  vessels  up  and  including  next  generation  aircraft  carriers  and 
submarines. 

NESM  will  materially  contribute  to  the  design  of  next  generation  naval  weapon  systems 
and  platfonns,  support  planning  and  rehearsal  of  ship  tests  prior  to  Life  Fire  Testing  (more 
“bang”  per  test  dollar),  and  the  evaluation  of  planned  (or  potential)  operational  use  scenarios. 
NESM  has  been  officially  adopted  by  the  Navy  for  these  uses.  “The  NAVSEA  Technical 
Warrant  (for  Shock/Ships)  concurs  that  NESM  is  the  appropriate  and  technically  acceptable 
modeling  and  simulation  (M&S)  tool  which  meets  the  M&S  requirements  to  support  current  and 
future  surface  ship  shock  applications.”  It  was  previously  approved  for  “Full  Ship  Shock  Trials 
(FSST)  Alternative  R&D  Programs  (PEO  Ships  &  PEO  Carriers),”  which  led  to  the  release  of 
OPNAVINST  9072. 2A  providing  future  ship  classes  w/  an  alternative  to  Full  Scale  Shock  Trials. 
NESM  has  been  used  to  support  Littoral  Combat  Ship  (LCS)  Live-Fire  Test  &  Evaluation 
(LFT&E)  and  the  USS  Cole  Validation  Study,  and  to  provide  support  for  Live  Fire  Test  and 
Evaluation  for  the  Navy’s  next  generation  Nuclear  Aircraft  Carrier  (CVN-78  and  79). 

Integrated  Hydrodynamic  Design  Environment  (IHDE)  — Facilitates  hydrodynamic 

performance  evaluations  for  early  stage  ship  design. 

IHDE  is  a  desktop  application  that  integrates  a  suite  of  Navy  hullform  design  and 
analysis  tools  allowing  a  user  to  perform  evaluations  of  performance,  including  visualization,  in 
a  simplified  and  timely  manner  from  a  single  interface.  Prior  to  the  development  of  IHDE,  naval 
architects  and  marine  engineers  often  had  to  learn  how  to  use  a  dozen  or  more  individual  design 
tools,  each  with  a  different  user  interface  and  input  format.  IHDE  provides  a  single  interface  for 
access  to  all  of  the  tools.  In  a  few  days  to  weeks,  a  single  user  with  IHDE  can  finish  projects  that 
took  several  highly  experienced  users  many  months  to  complete.  Current  capabilities  are  geared 
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toward  surface  ships,  both  monohulls  and  multihulls— including  catamarans  and  trimarans. 
Typical  uses  include  predicting:  1)  resistance  in  calm  water,  2)  seakeeping  behavior  in  waves,  3) 
hydrodynamic  loads  due  to  wave  slamming,  and  4)  operability  (percentage  of  time  a  ship  can 
carry  out  its  particular  mission  in  various  parts  of  the  world  based  on  historic  sea  state  data. 

The  US  Navy’s  Center  for  Innovation  in  Ship  Design  (CISD)  has  used  IHDE  to  assess 
the  performance  of  many  ship  designs,  including:  1)  T-AGOS-19  Ocean  Surveillance  ship;  2) 
Hospital  ship  (Mercy)  replacement  design;  3)  Salvage  Tow  &  Rescue  (T-STAR);  4)  Green 
Arctic  Patrol  Vessel  (GPAV);  5)  Medium  Affordable  Surface  Combatant  (MASC);  and  6)  an 
optimized  MASC.  IHDE  is  also  an  important  adjunct  capability  for  medium  fidelity  analysis  of 
ship  designs  developed  with  RSDE.  It  was  used  with  RSDE  as  part  of  the  Amphibious  Landing 
Craft  LX(R)  Analysis  of  Alternatives  and  the  Small  Surface  Combatant  Trade  study. 

SENTRi — Electromagnetic  Tools  for  DoD  Systems 

SENTRi  is  a  robust  and  high-fidelity  Full  Wave  electromagnetic  prediction  code  for  RF 
modeling  of  antennas,  microwave  circuits,  and  radar  cross-section  prediction.  SENTRi  is 
designed  for  the  modeling  of  complex  structures — including  highly,  heterogeneous  material 
structures  with  multi-scaled  features.  A  key  goal  is  the  calculation  of  the  simultaneous 
perfonnance  of  multiple-antenna  systems  embedded  on  a  platform.  The  key  features  for 
electromagnetics  are  based  on  solutions  of  Maxwell’s  equations  with  advanced  hybrid  finite- 
element  boundary-integral  techniques.  This  provides  high  accuracy  with  the  ability  to  solve 
large,  complex  problems.  SENTRi  is  continuously  validated  with  DoD  measurements.  SENTRi 
is  being  used  for  antenna  design,  antenna  in-situ  analysis,  RF  signature  prediction, 
Electromagnetic  Interference  (EMI),  Electromagnetic  Compatibility  (EMC),  material  modeling, 
microwave  device  analysis  (i.e.  waveguides,  filters,  circulators,  power  dividers),  phased  array 
antenna  systems,  and  apertures  (i.e.  radomes,  windows,  frequency  selective  surfaces).  SENTRi  is 
being  used  by  approximately  30  DoD  organizations  (government  and  industry). 

Capstone — Rapid  Geometry  and  Mesh  generation 

Capstone  is  a  CAD-neutral  application  that  provides  two  distinct  capabilities.  The  first 
is  the  capability  to  develop  numerical  representations  of  a  DoD  weapon  system  (i.e.  a 
Nonuniform  Rational  B-Splines  (NURBS)-based  digital  product  model  consisting  of  the 
platform  geometry  with  the  associated  attributes).  The  second  is  the  capability  to  generate  a 
mesh  from  the  geometry.  Valid  and  easily  produced  meshes  with  the  required  accuracy  are  the 
essential  starting  point  for  the  other  CREATE  (solver)  tools  for  detailed  analysis.  In  addition,  a 
number  of  non-CREATE  groups  use  Capstone  for  its  geometry  and  mesh  generation  capability 
for  their  applications. 

A  digital  product  model  has  many  advantages  for  acquisition.  It  enables  automated 
design  optimization.  It  facilitates  the  transfer  of  design  infonnation  between  the  government  and 
contractors,  eliminating  much  of  the  reliance  on  paper  documents,  and  improving  the  accuracy 
and  speed  of  information  flow.  It  provides  a  permanent,  analyzable  description  of  the  platfonn 
through  all  stages  of  the  acquisition  process.  Copies  of  the  product  model  can  be  generated  and 
assigned  to  individual  airplanes  allowing  the  DoD  to  track  the  history,  performance  and 
maintenance  of  entire  life  cycle  of  each  airplane.  Together  with  DaVinci,  which  builds  on  top  of 
the  Capstone  platform,  it  enables  the  recent  Air  Force  initiatives  of  the  Digital  Thread  and  the 
Digital  Twin[12]. 
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CREATE  Program  Organization  and  Management 

To  develop  and  deploy  the  CREATE  software  applications,  we  worked  with  the  DoD 
organizations  responsible  for  overseeing  the  design  and  analysis  of  air  vehicles,  naval  vessels 
and  RF  antennas  (Figure  2)  to  empower  them  to  develop  and  deploy  the  tools.  The  CREATE 
Program  first  fonned  four  projects  with  a  total  of  nine  multi-disciplinary  teams  of  subject  matter 
experts.  Then  they  identified  a  team  leader  within  a  DoD  customer  organization  for  each 
software  product  who  possessed  the  right  mix  of  subject  matter  and  high  performance  computing 
expertise;  and  leadership,  program,  and  project  management  skills.  They  helped  the  leader  build 
a  multidisciplinary  software  development  team  with  approximately  10  members.  Each  core 
development  group  is  located  at  a  customer  organization  (e.g.  the  Navy’s  ship  design  groups  are 
located  at  the  Naval  Surface  Warfare  Center,  Carderock  Division,  Bethesda,  MD).  Additional 
developers  are  drawn  from  other  organizations  as  needed  to  provide  expertise  not  available  in  the 
customer  organization.  For  instance,  the  structural  dynamics  modules  for  NESM  and  Kestrel  are 
being  supplied  by  the  Sierra  Mechanics  group  at  the  US  Department  of  Energy’s  Sandia  National 
Laboratory.  Generally  about  1/3  of  each  team  resides  at  the  core  organization,  and  the  remaining 
two-thirds  are  located  at  other  organizations.  The  CREATE  staffing  mix  is  about  45  DoD 
employees  and  about  75  DoD  support  contractors.  The  team  members  are  distributed  across  ~25 
collaborating  institutions. 

The  CREATE  Program  leaders  then  sponsored  each  team  by  providing  funding  and 
active  management  and  oversight  of  the  development  process.  They  developed  a  set  of  software 
project  management  and  software  engineering  practices  for  the  CREATE  Program  and 
promulgated  them  to  the  teams  as  guidance.  They  sought  a  balance  between  a  very  agile 
development  process  to  allow  the  teams  the  flexibility  to  accomplish  technically  difficult  tasks 
and  ensuring  adequate  accountability  together  with  an  organized  code  development  process.  The 
HPCMP  and  CREATE  Program  Office  actively  manages  the  nine  teams  cooperatively  with  the 
hosting  Service  organizations.  While  clear  lines  of  authority  and  obligation  have  been  formally 
established  between  the  HPCMP  CREATE  program  and  each  executing  and  hosting 
organization,  both  groups  have  developed  a  high  degree  of  trust  and  work  together  to  resolve 
conflicts.  There  is  a  strong  degree  of  alignment  on  the  technical  aspects  of  the  CREATE  Program 
between  both  groups. 

Each  project  has  a  Board  of  Directors  consisting  of  senior  members  of  the  relevant 
customer  organization  (Navy,  Air  Force  and  Army  acquisition  engineering  communities).  Each 
Board  meets  at  least  once  a  year.  The  Boards  review  the  progress  during  the  prior  year,  advise 
the  project  about  new  requirements,  and  serve  as  liaisons  between  the  CREATE  projects  and 
their  DoD  customer  community.  The  Board  members  are  members  of  the  Senior  Executive 
Service,  or  other  senior  staff  of  the  Navy,  Air  Force  or  Army. 

The  annual  CREATE  budget  is  about  $25M/year.  The  total  investment  in  CREATE  by 
the  DoD  HPCMP  from  2008  through  2015  is  ~  $160M.  The  Services  provide  “in-kind” 
contributions  of  another  ~$1  lM/year,  a  testament  to  the  value  of  CREATE  to  them.  The  Service 
contributions  include  office  space  and  supplies;  admin  support;  network  and  other  host  services; 
additional  professional  staff  for  the  code  development  teams;  and  access  to  validation 
experiments  and  data. 
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Figure  2  CREATE  Program  Organization  Chart 


CREATE  Builds  the  Right  Software  and  Builds  It  Right 

The  CREATE  Program  developed  a  vision  for  how  DoD  could  implement  virtual 
prototypes  with  physics-based  high  performance  computing  engineering  software  within  its  own 
processes  to  “revolutionize”  its  lethargic  acquisition  process.  It  then  fleshed  out  that  vision 
through  joint  assessments  with  each  service  of  their  detailed  acquisition  processes  to  identify  the 
specific  tools  needed  to  reduce  the  time,  cost,  and  risk  and  improve  the  system  performance  for 
the  Service’s  acquisition  programs.  This  vision  is  described  in  an  “Initial  Capability  Document” 
(ICD)  for  each  project.  The  ICDs  were  reviewed  and  approved  by  the  Board  of  Directors  for 
each  project  and  are  reviewed  periodically. 

Although  CREATE  was  proposed  to  be  a  12  year  program,  the  CREATE  tools  are 
designed  for  a  30  to  40  year  life  since  that  is  the  expected  life  span  of  successful  engineering 
codes.  Each  CREATE  code  development  team  follows  a  highly  disciplined  software 
development  process.  There  is  a  strong  emphasis  on  software  quality.  To  facilitate  development 
by  non-collocated  teams,  the  CREATE  Program  provides  a  supportive  code  development 
environment  with  virtual  clusters,  central  servers  with  code  and  document  repositories,  issue 
trackers,  user  and  developer  forums,  configuration  management  services,  and  access  to  high 
performance  computers  for  testing  and  performance  enhancement,  and  high  quality  video 
conferencing.  Guided  by  the  ICD,  a  12-year  product  roadmap  and  feedback  from  the  BODs  and 
customers,  each  product  team  issues  a  new  release  each  year  with  the  upgraded  capability  and 
new  features  needed  by  the  customer  communities.  This  places  the  upgraded  tool  into  the  hands 
of  the  user  community,  and  gives  the  code  development  team  rapid  feedback  on  the  quality  and 
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usefulness  of  the  upgraded  code.  The  annual  release  cadence  (Figure  3)  adds  discipline  to  the 
development  process. 

The  CREATE  teams  generally  use  a  SCRUM  “agile”  development  process  tailored  for 
their  environment  and  code.  The  releases  are  designed  and  tested  for  all  the  standard  Linux  and 
Unix  operating  systems.  Each  release  is  extensively  tested  both  during  the  development  process, 
and  during  the  release  process. 

Each  software  release  is  documented  with  a:  l)Product  Technical  Description  that 
describes  the  physics  and  engineering  capabilities  (including  the  equations)  in  the  code,  the 
computer  science  approaches,  and  the  solution  algorithms;  2)  Developers  Manual  that  describes 
the  source  code  in  detail,  provides  an  index/table  of  contents,  and  other  information  essential  for 
understanding  the  source  code;  3)  Users’  Manual  to  help  the  users  set  up  their  problems,  run  the 
code,  and  analyze  the  results;  and  4)  test  plan  with  an  archive  of  test  problems  with  the  input  and 
test  results.  In  addition  there  are  tutorials  and  a  user  forum  on  the  CREATE  server,  all  backed-up 
by  a  user  support  group. 


Figure  3.  CREATE  Annual  Releases  with  version  numbers.  The  FY2015  4th  quarter  and 
FY2016  releases  are  planned.  Some  CREATE  products  had  releases  prior  to  201 1. 
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Information  Assurance  for  the  CREATE  products 

Information  assurance  for  computational  engineering  applications  can  be  understood 
from  its  role  in  the  high  perfonnance  computing  (HPC)  ecosystem.  The  CREATE  tools  are  part 
of  an  ecosystem  consisting  of  1)  Subject  Matter  Experts  (SMEs)  who  use  the  tools  on  high 
perfonnance  computers  over  high  speed  networks  to  generate  virtual  prototypes  and  analyze 
their  performance;  2)  software  applications  like  the  CREATE  tools  that  can  be  used  to  generate 
virtual  prototypes  and  predict  their  perfonnance;  3)  Experimental  testing  organizations  to 
generate  validation  data  to  establish  the  validity  of  the  models  that  are  the  basis  of  the  software 
applications;  4)  high  speed  computer  networks  that  provide  the  SMEs  access  to  supercomputers; 
5)  supercomputers;  and  6)  sponsors  who  need  the  results  of  the  calculations  and  provide  the 
funds  to  generate  the  results. 

CREATE  is  designed  to  provide  the  DoD  a  military  competitive  advantage.  Thus  the 
DoD  must  control  the  distribution  and  use  of  the  codes.  The  CREATE  codes  are  unclassified,  but 
are  subject  to  the  International  Traffic  in  Arms  Regulations  (ITAR).  To  be  effective,  DoD  users 
must  be  able  to  run  the  CREATE  codes  securely  and  access  their  proprietary  data  on  a  high 
perfonnance  computer.  The  codes  and  data  are  encrypted  at  rest  and  accessed  and  transferred 
over  a  secure  encrypted  network  with  two-factor  authentication.  To  ensure  integrity  of  the  codes 
and  the  users’  data,  the  codes  and  the  user’s  data  are  archived  on  the  HPCMP  supercomputers 
and  backed  up  frequently  to  a  remote  data  repository. 

However  many,  if  not  most,  DoD  engineers  have  access  only  to  a  Windows  Personal 
Computer  with  Microsoft  Office  and  a  Browser.  Usually  no  other  software  is  allowed  so  those 
engineers  cannot  access  the  HPCMP  supercomputers  and  the  CREATE  tools.  To  remove  this 
barrier  to  access,  the  DoD  HPCMP  CREATE  program  has  developed  a  “portal”  that  allows  DoD 
users  to  access  the  DoD  HPCMP  supercomputers  through  their  browser.  The  “portal”  features 
two-factor  authentication  and  encrypted  data  transfer.  It  allows  users  to  set  up  their  job;  run  it; 
and  store,  analyze,  and  visualize  the  results  through  their  browser. 

To  prevent  unauthorized  access  to  the  CREATE  codes  and  to  the  intellectual  property  of 
the  users,  we  limit  access  to  codes  to  DoD  employees  or  DoD  contractors  who  have  a  valid 
reason  for  access  to  the  CREATE  software.  They  must  sign  a  software  distribution  agreement 
that  describes  the  limitations  of  their  use  (not  to  redistribute  the  code,  reverse  engineer  it,  etc.) 
and  their  intended  use.  They  also  agree  to  abide  by  the  ITAR  procedures  which  have  civil  and 
criminal  penalties  if  violated.  The  CREATE  source  code  is  only  accessible  to  the  development 
team.  The  ideal  is  a  “Software  as  a  Service”  model  where  the  user  can  only  execute  the  code,  but 
not  get  a  copy,  even  an  executable.  However,  that’s  not  practical  for  some  users. 

Intellectual  Property  Rights 

The  DoD  HPCMP  must  have  “government  purpose  rights”  to  be  able  to  distribute  the 
CREATE  software  to  users.  Even  a  single  line  of  code  for  which  the  DoD  doesn’t  have  these 
rights  would  leave  the  DoD  vulnerable  to  lawsuits  and  large  financial  settlements  for  copyright 
or  patent  infringement  and  theft  of  intellectual  property.  Legal  reviews  of  the  ~  40  CREATE 
contracts  have  detennined  that  the  DoD  does  have  “government  purpose  rights.”  Remedial 
action  was  necessary  for  some  of  the  contracts. 

It  is  DoD  policy  to  share  DoD  RDT&E  results  with  the  US  Defense  Industry  if  it  is  in  the 
interests  of  the  DoD  because  a  strong  US  defense  industry  is  essential  for  national  security[13]. 
Several  defense  industries  have  expressed  a  strong  interest  in  adopting  the  CREATE  tools  for  use 
in  their  commercial  as  well  as  military  design  work.  This  requires  that  the  DoD  have 
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“unrestricted  rights”  for  the  relevant  CREATE  software.  As  the  result  of  further  extensive  legal 
reviews  of  the  CREATE  contracts  over  the  last  three  years  together  with  additional  remedial 
action,  the  DoD  now  has  “unrestricted  rights”  to  the  CREATE  tools.  We  expect  that  by  early 
2016,  several  large  Defense  Companies  will  be  using  CREATE  tools  for  both  military  and 
commercial  systems. 

Transition  and  Adoption 

Transition  of  research  results  to  applications  and  products  has  been  a  historic  challenge 
for  much  of  the  Science  and  Technology  research  done  by  the  DoD  research  community.  The 
approach  adopted  by  the  CREATE  Program  of  embedding  the  CREATE  development  teams  in 
the  DoD  customer  organizations  responsible  for  the  design  and  development  of  the  relevant 
weapon  system  has  been  very  successful  in  overcoming  that  challenge.  The  CREATE  teams  are 
trusted  agents  of  their  DoD  organization.  For  example,  to  assess  various  options  for  a  follow-on 
to  the  Littoral  Combat  Ship,  or  develop  a  concept  for  the  Ohio  Replacement  Submarine,  the 
Navy  turned  to  the  trusted  organizations  responsible  for  those  tasks,  the  Naval  Architecture  and 
Engineering  Department  at  the  Naval  Surface  Warfare  Center  at  Carderock.  These  were  the  same 
groups  developing  the  ship  design  tool,  RSDE  and  the  hydrodynamics  tool  NavyFOAM.  The 
transition  was  almost  automatic. 

In  general,  computational  design  tools  are  very  effective  methods  for  capturing  the 
corporate  knowledge  and  new  research  results  in  a  field,  and  giving  design  engineers  access  to 
that  knowledge  in  a  tool  that  the  engineer  can  use  to  design  a  new  system.  This  greatly  facilitates 
the  development  of  innovative  designs.  It  also  gives  the  engineer  an  opportunity  to  compare  the 
impact  of  new  research  results  in  the  context  of  present  practice,  and  allows  the  engineer  to 
answer  many  “What  if?”  questions.  This  has  caught  the  attention  of  a  number  of  groups  in  the 
DoD  Science  and  Technology  research  community.  Those  groups  are  beginning  to  work  with  the 
CREATE  team  to  incorporate  their  research  results  into  the  CREATE  codes  as  a  means  to 
transition  their  research  results  to  acquisition  programs. 

Summary  and  Future 

The  CREATE  program  is  a  set  of  physics-based  high  performance  computing  software 
tools  that  enable  the  DoD  to  develop  innovative  systems.  The  tools  enable  DoD  engineers  to 
generate  and  analyze  virtual  prototypes  of  DoD  Air  Vehicles,  Ships,  and  RF  antennas,  and  in  the 
future,  Ground  Vehicles,  and  accurately  predict  their  performance.  The  CREATE  tools  are: 

1)  Government-developed,  government-owned,  and  government-supported  to  enable  the  DoD  to 
independently  evaluate  contractor  designs  and  other  deliverables;  2)  Designed  for  a  ~30  year+ 
life  cycle;  3)  Being  adopted  by  DoD  acquisition  engineering  communities  (government  and 
industry);  and  4)  Are  beginning  to  have  significant  impact  on  acquisition  programs. 

While  CREATE  was  proposed  as  a  12  year  program,  it  is  proving  to  be  highly 
successful  and  very  promising  after  only  8  years..  In  addition,  there  are  many  more  opportunities 
for  tools  like  CREATE  in  other  areas.  CREATE  Ground  Vehicles  was  started  in  2013.  Recently 
two  Army  and  Air  Force  hypersonics  groups  have  requested  that  CREATE  upgrade  the  fixed- 
wing  aircraft  tool,  Kestrel,  to  handle  hypersonic  flight.  Other  potential  topic  areas  include  rocket 
and  missile  design;  satellite  operations;  structural  design  for  ships  and  air  vehicles;  application  to 
improving  the  DoD  Test  and  Evaluation  Process;  lifetime  prediction  for  ships,  air  vehicles  and 
ground  vehicles;  incorporating  life  cycle  costs  in  the  design  concept  tools  RSDE  and  DaVinci; 
continuing  support  for  the  DoD  Engineered  Resilient  Systems  Program 
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(http://www.defenseinnovationmarketplace.mil/resources/ERS_COI_March_20 1 5 .pdf),  the  Air 
Force  Digital  Thread  and  Digital  Twin  Programs[12];  and  similar  programs  by  the  Navy  for 
ships  and  the  Army  for  ground  vehicles. 
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